feat: OpenRouter video surface drift leg (cost-safe live models canary) - #323
Merged
Conversation
Adds the first net-new live drift surface beyond the original three (OpenAI/Anthropic/Gemini). The OpenRouter async video-generation proxy (src/openrouter-video.ts, video-proxy-shared.ts) shipped on master with zero drift coverage; this closes that gap without incurring any paid video generation. - New surface slug `openrouter-video` in surface-registry.ts pointing at the OpenRouter video handlers, so the collector routes its drift blocks. - Live canary (FREE, metadata only): listOpenRouterVideoModels hits GET /api/v1/videos/models and asserts the model families aimock mirrors (seedance, sora) are present. Gated on OPENROUTER_API_KEY. - Static envelope-shape legs (no key, run in CI): triangulate the submit, completed-status, and models-listing envelopes the proxy emits against hand-authored exemplars — exercising the real handler + collector path. - Wire OPENROUTER_API_KEY into the drift job env in test-drift.yml and fix-drift.yml so the live canary runs once the secret is mirrored.
commit: |
jpr5
marked this pull request as ready for review
July 22, 2026 23:37
jpr5
added a commit
that referenced
this pull request
Jul 23, 2026
…ed) (#327) ## What Adds a **live** queue-lifecycle drift probe for the fal.ai queue surface (`src/fal.ts`). Rows 25/26 of the drift-coverage audit were STATIC-only; this closes the live gap for the queue surface with a **cost-safe** canary that self-activates once `FAL_KEY` is mirrored to repo secrets. Mirrors the OpenRouter-video precedent (#323): live leg is `describe.skipIf(!FAL_KEY)`, so it **skips in CI until the secret lands**, then runs automatically. ## Changes - **`src/__tests__/drift/providers.ts`** — new `falQueueLifecycleCanary(apiKey, modelId, input)`: drives the REAL fal queue API through **submit → status → immediate cancel**, returning the three envelope bodies. Cancel is issued even if the status poll throws (never leaves a job to RUN). - **`src/__tests__/drift/fal-queue.drift.ts`** — new `describe.skipIf(!FAL_KEY)("fal.ai queue lifecycle (live, cost-safe)")` block: asserts the load-bearing submit/status/cancel fields on the real envelope and triangulates each step (exemplar × real × aimock mock) under the `fal-queue` surface slug. Completed-**result** envelope stays STATIC-only (never fetched — that is the only paid retrieval). - **`.github/workflows/test-drift.yml` + `fix-drift.yml`** — add `FAL_KEY: ${{ secrets.FAL_KEY }}` to every drift-job env block, alongside the existing provider keys (minimal/additive). Surface registry already contains the `fal-queue` slug — no registry edit needed. ## Cost note fal bills **compute only when a queued job RUNS**. The canary submits (free) and cancels immediately while the job is still `IN_QUEUE`, so **expected cost is \$0**. It never fetches the result endpoint. Cheapest reliably-available model (`fal-ai/flux/schnell`) bounds the worst case; **residual exposure is at most one sub-cent generation** in the rare race where the model completes before the cancel lands. ## Red-green Live red-green cannot run locally (`FAL_KEY` is not yet a repo secret and 1Password access is guard-blocked) — **CI will capture the live red-green once `FAL_KEY` is mirrored.** Structural red-green proven locally against the real drift machinery: **RED** — forced the live block to run with a deterministically drifted real submit envelope (`queue_position` as string vs the mock's number): ``` API DRIFT DETECTED: fal.ai queue submit (live) Surface: fal-queue 1. [critical] TYPE MISMATCH between real API and mock: string vs number Path: queue_position SDK: number Real: string Mock: number Test Files 1 failed (1) ``` **GREEN** — reverted; live block skips (no `FAL_KEY`), static legs pass: ``` ✓ src/__tests__/drift/fal-queue.drift.ts (5 tests | 1 skipped) Test Files 1 passed (1) ``` Full suite: `Test Files 16 passed | 9 skipped (25) Tests 93 passed | 65 skipped`. Typecheck (touched files), prettier, eslint, build all clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this covers
The first net-new live drift surface beyond the original three (OpenAI /
Anthropic / Gemini). The OpenRouter async video-generation proxy
(
src/openrouter-video.ts,video-proxy-shared.ts) is shipped onmasterbuthad zero drift coverage. This adds the
openrouter-videosurface:openrouter-videoslug insurface-registry.tspointing at the OpenRouter video handlers, so the drift-report collector
routes any drift block from this leg to the right source file (exit-2
auto-fix lane) instead of quarantining it.
listOpenRouterVideoModelsauthenticateswith
OPENROUTER_API_KEYand hits the public listing endpointGET /api/v1/videos/models, asserting the video-model families aimock'sproxy mirrors (
seedance,sora— theDEFAULT_OPENROUTER_VIDEO_MODELSset) are still present. Gated with
describe.skipIf(!OPENROUTER_API_KEY).over HTTP and
triangulatethe submit response, completed status-poll, andmodels-listing envelopes the proxy emits against hand-authored exemplars.
These exercise the real handler + triangulate + collector routing path,
not a unit fake.
OPENROUTER_API_KEYadded to the drift-jobenv:inboth
.github/workflows/test-drift.ymland.github/workflows/fix-drift.yml.Cost-safe scoping rationale (no paid generation)
Video generation costs real money per job, so nothing here submits a
generation. The daily live-reality check is the models listing endpoint
(free, metadata only). Job-lifecycle envelope shapes are asserted statically
against the mock's own output — no live paid call. A live job would be the only
way to triangulate the content download against reality; that is deliberately
deferred as a follow-up rather than incurring generation cost. Grok Imagine
(xAI) and Veo (Google) live canaries are also deferred — their listing
endpoints live on different hosts and need their own keys.
Red-green proof
The real failure surface is the drift collector / triangulation, not a unit
fake. Structural red-green was captured locally against
pnpm run test:drift.RED — the mock envelope was deliberately diverged by renaming
supported_resolutions→resolutionsinhandleOpenRouterVideoModels(
src/openrouter-video.ts), then running the leg:The failure carries the
Surface: openrouter-videomarker, so the collectorroutes it structurally — proving the whole triangulate → formatDriftReport →
collector path is exercised.
GREEN — reverted the one-line divergence, re-ran:
Full suite also green:
pnpm run test:drift→ 16 passed | 9 skipped files;drift-collector.test.ts(registry coverage/integrity) → 160 passed;eslint,prettier --check, andtsdownbuild all clean.Live validation dependency (deferred to CI)
The live canary's red-green is deferred to CI:
OPENROUTER_API_KEYis notavailable in the local agent environment, so faking it was not an option. Once
the secret is mirrored to repo secrets, the
drift-live-prjob (and the dailydriftjob) will run the live leg. Until then the live canarydescribe.skipIf-skips in CI — that is expected, not a failure. This has notbeen claimed live-verified; only the structural red-green was run locally.
Notes
src/__tests__/drift/*+.github/workflows/*only — not in the publishednpm package, so no version bump / CHANGELOG.
🤖 Generated with Claude Code